Adapting graphical data, processing activity to changing network conditions

ABSTRACT

The performance of distributed systems can be improved by adapting a server&#39;s processing activity in response to changing network conditions or performance mismatches. In one embodiment, the server&#39;s processing rate is modified when a change in the network conditions is detected. In another embodiment, the encoding technique used to process the graphical data is modified upon detecting a change in the network conditions. In yet another embodiment, the server&#39;s processing rate is modified upon detecting a performance mismatch between the client and the server. A time differential relating to the transmission and receipt of graphical data between the server and the client is used to estimate the network conditions. Similarly, another time differential relating to the processing of graphical data by both the client and server is used to ascertain performance mismatches between the client and server.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This claims priority to and the benefit of U.S. provisionalpatent application No. 60/207,532, filed May 26, 2000, and 60/225,217,filed Aug. 14, 2000, the entirety of which are incorporated herein byreference.

FIELD OF THE INVENTION

[0002] The present invention relates generally to communication betweenan application server and a thin client in a distributed system and, inparticular, to the reduction in the amount of graphical display datatransmitted over a low bandwidth transport protocol mechanism.

BACKGROUND OF THE INVENTION

[0003] Distributed computer systems utilize the technique ofdistributing application execution. More specifically, an applicationserver locally executes an application program and provides theapplication output data to clients/network users who then display theresults on a display screen coupled to their local computer. Byperforming the application processing functions at the server anddisplaying the application output at the client, these distributedsystems make the best use of server and client resources, i.e., the morecapable server performs the compute and memory intensive applicationprocessing functions while the client, with potentially greater videoperformance, displays the output. During execution of the applicationprogram, a user of the client views the application output data on theclient's display and interacts with the application program via keyboardor mouse inputs. The client user's inputs correspond to requests to theapplication server to perform certain actions that affect the operationof the application program. The application server passes these requeststo the application for further processing. The application server alsointercepts the application output data, generated by the applicationprogram and typically targeted to the operating system of theapplication server, and transmits all or part of the data to the clientfor display. From the perspective of the user, the application programseems to be executing locally even though it is actually being executedon a remote server and just being displayed locally.

[0004] The performance of distributed systems depends to a large degreeon the capability of the network, which couples the client and theserver, and therefore on the amount of graphical data (corresponding tothe application output data) that is transmitted over the network. Sincethe application server and the client workstation are generally linkedonly by a transport mechanism (such as serial lines, telephone lines,local area networks and wide area networks) that possesses lowerbandwidth than would be otherwise available if the application programwere executing locally, the perceived performance of the distributedsystem may suffer. Accordingly, manufacturers of distributedsystems/software continually strive to improve the performance of suchsystems.

SUMMARY OF THE INVENTION

[0005] The present invention improves the performance of distributedsystems by reducing the amount of graphical data (e.g., bitmappedgraphical data, encoded bitmaps, glyphs, and line data) transmittedbetween the server and the client. The invention achieves this reductionin several different ways, for example, by encoding the graphical datainto a smaller object, by representing a graphical object with indiciaof the object, by increasing the repetitiveness of the data in theprotocol stream so that compression algorithms operate more efficiently,by tracking and leveraging the prior transmissions ofidentical/repetitive graphical objects, by adapting the rate ofprocessing activity or the encoding technique in response to changes inthe network performance or in response to performance mismatches betweenthe client and server, and in several other ways described herein.

[0006] In one embodiment, the present invention enables the efficienttransmission of graphical display data from an application server nodeto a client node. In this embodiment, the invention intercepts aplurality of graphical display information commands issued by anapplication executing on the server. Once these commands have beenintercepted or otherwise obtained, they are analyzed to determinewhether they should be selected for further transmission to remoteclient nodes. The selected commands are then encoded and packaged into aprotocol packet stream for eventual transmission to the remote clientnodes. Prior to transmission, the packet stream can be furthercompressed by subjecting the encoded packets to compression techniques.

[0007] The encoding technique of the present invention substantiallymaximizes compression of the protocol packet stream by building inrepeatability into the protocol stream that allows the compressionalgorithm to substitute pointers for repetitive data. This techniquefocuses on maximizing the efficiency of the compression algorithm,rather than solely reducing the amount of data via encoding prior totransmission. The result of this encoding scheme is that the degree ofcompression is significantly increased for many types of graphical data.The encoding technique of the present invention encodes the graphicaldata so that portions of the graphical data that repeat are encoded inthe same manner. The compression algorithm then operates on thisrepetitive encoded data to realize a greater degree of efficiency.

[0008] In one embodiment, the present invention efficiently reduces theamount of repetitive graphical data transmitted to a remote displaydevice coupled to the client. In this embodiment, a server agentsearches for an index (that may have been previously stored on theserver by the server agent) associated with the graphical data and whichis indicative of a prior transmission of the graphical data to a clientagent. The index or handle represents a memory location in the client'scache memory that currently stores the previously transmitted graphicaldata. If the index is found, then the index is transmitted to the clientagent, thus avoiding the need to transmit the graphical data itself. Theclient agent obtains the graphical object stored in the cache locationidentified by the index and performs additional processing activities asdirected by the server agent. In one aspect of the invention, commandsand information transmitted between the server agent and the clientagent are queued at the server. The commands in the queue can then beprocessed in accordance with an estimate of the relative performancebetween the client and server agents or computers.

[0009] If the server agent is unable to locate the index, the serveragent searches a fuzzy database for a fuzzy key that is associated withthe graphical data. The fuzzy key, for example, corresponds to a lengthof the graphical data. The fuzzy key indicates that the graphical datahad at one time been transmitted to the client and that there is astrong likelihood that the graphical data is stored within the client'spersistent storage memory. One or more fuzzy keys are provided to theserver agent upon initiating a session between the client and serveragents. Alternatively, the fuzzy keys are transmitted to the serveragent in accordance with a predetermined polling interval. Regardless ofhow the fuzzy keys are received, the server agent stores them in thefuzzy database for subsequent use during the client/server session. Thefuzzy database relating to a particular session can also be maintainedbeyond the term of the session so that it is available for futuresessions, by, for example, having the client agent persistently cachethe graphical data from the particular session and causing the serveragent to dynamically construct the fuzzy database on the server from theclient's persistently-cached graphical data during a future session.This may prove useful in environments where frequent, but short,sessions are present.

[0010] If the fuzzy key is located, then the server agent transmits thefuzzy key together with a command to the client agent, which, forexample, retrieves the graphical data stored in its persistent storagememory and copies it into its volatile memory. If the client agent failsto retrieve the graphical data, the server agent can transmit thegraphical data, together with a recovery marker, to the client agent.

[0011] If the server agent is unable to locate the fuzzy key within thefuzzy database, the server agent forms the fuzzy key and stores thefuzzy key in the fuzzy database. The server agent then transmits thefuzzy key, an index, and the graphical data to the client agent, whichis instructed to store the data in the client's volatile memoryassociated with the index. As the client-server session continues andthe client's volatile memory becomes full, graphical data will betransferred into the client's persistent storage. In one aspect theclient agent stores the graphical data in a file residing in thepersistent storage memory that reflects the fuzzy key as part of thefile's name.

[0012] In one aspect of the invention, the server agent instructs theclient agent to update an off-screen surface coupled to the client agentusing the graphical data identified by the server agent and/or stored inthe client's volatile or persistent storage memory. These instructionscan be triggered in response to a change in the off-screen surface, suchas when the off-screen surface and on-screen surface of the client nodeinteract. If the client agent fails to update the off-screen surface inaccordance with the server agent's instructions, the server agent cantransmit the graphical data to the client agent and direct that theclient agent update portions of its on-screen or off-screen surfaces,which may have been corrupted, with the transmitted graphical data.

[0013] Further, the encoding technique used to encode the graphical dataprior to transmission can be selected based on an estimate of thenetwork conditions of the network that couples the client agent with theserver agent. This estimate can correspond to a time differentialmeasured by the client agent that relates to the amount of time expendedto receive a particular encoded graphical data.

[0014] In one embodiment, the present invention adapts a processing ratein response to changing network conditions. In one aspect of theinvention, the server agent processes graphical data addressed to theclient agent at a first rate. By determining the network conditions ofthe network that couples the client and server agents, the server agentcan adjust its processing rate from the first rate to a second rate inresponse to a change in the network conditions. The server agent candetermine information about the network conditions by transmitting theprocessed graphical data to the client agent and instructing the clientto measure a time differential associated with the transmission orreceipt of the graphical data. In this manner, the time differentialprovides an indicator of the network conditions and the server agent canrationally select the second rate in accordance with this timedifferential. Similarly, the encoding technique used while processingthe graphical data by the server agent at the first rate can be modifiedto a second encoding scheme/technique in response to changes in thenetwork conditions that are determined by the server agent. The networkconditions can be estimated in accordance with the time differentialdiscussed previously. Once the second encoding technique is selected bythe server agent, the server agent will process subsequent graphicaldata using this second encoding technique.

[0015] In one embodiment, the invention adapts a processing rate of theserver in response to a performance mismatch between the server and theclient coupled to the server via the network. In operation, the serveragent processes graphical data and determines a first time periodassociated with such processing. For example, the first time period canbe determined by scrolling a frame buffer of the server. The clientagent also processes the graphical data and determines a second timeperiod associated with its processing. The server agent then determinesthe time differential between the first and second time periods andadjusts its processing rate in accordance therewith.

[0016] In one embodiment, the invention efficiently reduces the amountof graphical line data transmitted from a server to a client via acommunications network. The server agent separates a path defining aline or region into a plurality of strips, where each of the strips havea beginning and an endpoint coordinate defined within a coordinatesystem (e.g., a Cartesian coordinate system). The coordinate systemcorresponds to a region of a display surface associated with the client.In one aspect of the invention, the server agent quantizes thecoordinate system into a plurality of quantized angles (e.g., 16quantized angles) and determines the endpoint coordinate of one of theplurality of strips. The coordinate system is then normalized so thatits origin coincides with this endpoint coordinate. The beginningcoordinate of the adjacent strip in the path is then associated by theserver agent with the endpoint coordinate of the first strip. The serveragent then selects one of the quantized segments of the coordinatesystem as corresponding to an approximate angle of the adjacent strip.This technique enables the transmission of the difference between theendpoint coordinates of the two strips(i.e., the strip length) and anindication of the selected quantized angle and thus avoids having totransmit both sets of coordinates or a coordinate and the precise anglefor each strip.

[0017] In one embodiment of the invention, the server agent remotelycontrols the formation of an off-screen surface at a client, which iscoupled to the server via a communications network. The server agentinforms the client agent that an off-screen surface of a particular sizeor type is to be formed within the client's memory and the client agentresponds by selecting and allocating a first memory region having anappropriate format to represent the off-screen surface from a pluralityof available memory regions in the client's memory. The server agentthen either transmits a bitmap or an indicia of the bitmap (e.g., indexor fuzzy key) to the client agent and instructs the client agent to copythe bitmap to a particular location within the first memory region. Oncethe bitmap is copied to the off-screen surface, it can be used to updatethe client's on-screen surface. Such off-screen operations can also beperformed for non-bitmap data, such as lines or text.

[0018] The server agent can also respond to error conditions by storinga duplicate of the client's off-screen surface in the server's localmemory. In the event of an error condition, such as when a preselectedmemory region of the client's memory is not sufficiently large enough tostore a particular off-screen surface, the client agent asynchronouslyinforms the server agent of the error. In response to the errorcondition, the server agent transmits at least one portion of itsduplicate off-screen surface or a copy of the bitmap to the client agentand instructs the client agent to update the client's on-screen surfaceusing the transmitted duplicate off-screen surface data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The foregoing discussion will be understood more readily from thefollowing detailed description of the invention, when taken inconjunction with the accompanying drawings, in which:

[0020]FIG. 1 schematically illustrates a client node coupled to a servernode via a communication network, where the client and server nodes eachhave respective client and server agents operating in accordance with anembodiment of the present invention;

[0021]FIG. 2 provides a high-level flow diagram of the steps performedby the server agent of FIG. 1 in accordance with an embodiment of thepresent invention;

[0022]FIG. 3 provides a more detailed flow diagram to the diagram ofFIG. 2 in accordance with an embodiment of the present invention;

[0023]FIG. 4 schematically illustrates the contents of the client andserver's volatile and persistent memory, as affected by operations ofthe client and server agents of FIG. 1, in accordance with an embodimentof the present invention;

[0024]FIG. 5 provides a flow diagram of the steps taken to minimize theamount of data transmitted to the client by using caching techniquesinvolving a fuzzy database and fuzzy keys, in accordance with anembodiment of the present invention;

[0025]FIG. 6 schematically illustrates a set of procedures used toremote text information generated by an application program on a serverto a display of a distant client in accordance with an embodiment of thepresent invention;

[0026]FIG. 7A provides a line diagram illustrating a plurality ofsegments forming a particular path;

[0027]FIG. 7B provides a pixel-level representation of the line diagramof FIG. 7A, illustrating how the path is actually approximated on acomputer display;

[0028]FIG. 8 provides a flow diagram of the steps taken when processingand encoding segment and strip data in accordance with an embodiment ofthe present invention;

[0029]FIG. 8B provides a coordinate system organized into a set of 16regions that is used to quantize angle information relating to stripsand which improves compression efficiency in accordance with anembodiment of the present invention;

[0030]FIG. 9 superimposes the coordinate system of FIG. 8B on thepixel-level representation of FIG. 7B to illustrate an embodiment of astrip processing technique in accordance with the present invention;

[0031]FIG. 10 provides a table, which illustrates how to get to the nextstrip from the endpoint of the previous strip, for strips in each of thequantized segments of FIG. 8B in accordance with an embodiment of thepresent invention; and

[0032]FIG. 11 provides a flow diagram of the steps performed by theserver agent in response to an error condition involving the formationof a client's off-screen surface.

DETAILED DESCRIPTION OF THE INVENTION

[0033] The present invention reduces the amount of graphical datatransmitted between an application server, which is executing anapplication program, and a subscribing client, which is locallydisplaying the output data of the application program, by encoding thedata into relatively small representations that repeat within theprotocol stream transmitted between the client and the server. In thismanner, the invention minimizes the size of each discrete data elementthat must be transmitted and increases the repeatability of the datawithin the protocol stream so that compression algorithms that operatemore efficiently on repetitive encoded data can realize a greater degreeof compression efficiency. The invention uses a number of techniques torealize this enhanced compression, including the following: scanning acommand queue for later-issued commands which supercede earlier-issuedcommands; disk-caching techniques that improve compression of data thathas been previously encountered during a client-server session;transmission of relative coordinates using quantized regions to avoidsending both endpoints and/or the angle of an associated strip; andmanipulation of off-screen surfaces to enhance performance during theclient-server session. The invention also reduces the frequency ofoverscroll problems encountered when there is a performance mismatchbetween a fast server and a relatively slow network or client.

[0034] With reference to FIG. 1, one or more client nodes 110′, 110″(hereinafter each client node or plurality of client nodes is generallyreferred to as 110) are in communication with one or more server nodes150′, 150″ (hereinafter each server node or plurality of server nodes isgenerally referred to as 150) via a communications network 140. Thenetwork 140 can be a local-area network (LAN), a medium-area network(MAN), or a wide area network (WAN) such as the Internet or the WorldWide Web. Users of the client node 110 can be connected to the network140 through a variety of connections including standard telephone lines,LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadband connections(ISDN, Frame Relay, ATM), and wireless connections. The connections canbe established using a variety of communication protocols (e.g., TCP/IP,IPX, SPX, NetBIOS, and direct asynchronous connections). In someparticularly advantageous embodiments, the protocol used may be theIndependent Computing Architecture protocol manufactured by CitrixSystems, Inc. of Fort Lauderdale, Florida or the Remote Display Protocolmanufactured by Microsoft Corporation of Redmond, Washington.

[0035] The client node 110 can be any personal computer (e.g., 286, 386,486, Pentium, Pentium II, Macintosh computer), Windows-based terminal,Network Computer, wireless device, information appliance, RISC Power PC,X-device, workstation, mini computer, main frame computer or othercomputing device that has a windows-based desktop and sufficient memoryfor executing a small, display presentation program. The displaypresentation program uses commands and data sent to it by theapplication server 150 across the network 140 to render a graphicaldisplay. In a distributed computer system model, the execution ofapplication programs occurs primarily on the application server 150 andthe user interface, keystrokes, and mouse movements produced byclient-side inputs are transmitted over the network 140 to and from theclient node 110.

[0036] The client node 110 typically includes a processor 112, avolatile memory 114 (e.g., RAM cache), an operating system 116, a clientagent 118, a persistent storage memory 120, a network interface 122(e.g., a network interface card), a keyboard 124, a mouse 126, and adisplay 128. Windows-oriented platforms supported by the client node 110can include, without limitation, WINDOWS 3.x, WINDOWS 95, WINDOWS 98,WINDOWS NT 3.51, WINDOWS NT 4.0, WINDOWS 2000, WINDOWS CE, MAC/OS, Java,and UNIX. The client agent 118 corresponds to a software program thatreceives commands and data from the server node 150 and from a user (notshown) of the client node 110. The client agent 118 uses the receivedinformation when interacting with other components of the client node110 (e.g., when directing the operating system 116 to output data ontothe display 128). The client agent 118 also transmits requests and datato the server node 150 in response to server-issued commands or useractions at the client node 110.

[0037] Similarly, the server node 150 includes a processor 152, avolatile memory 154, an operating system 156, an application program158, a server agent 160, persistent storage memory 162, and a networkinterface 164 The server agent 160 corresponds to a software programthat interfaces with the client agent 118 and other components of theserver node 150 to support the remote display and operability of theapplication program 158.

[0038] Each application server 150 hosts one or more applicationprograms 158 that can be accessed by the client nodes 110. Examples ofsuch applications include word processing programs such as MICROSOFTWORD and spreadsheet programs such as MICROSOFT EXCEL, both manufacturedby Microsoft Corporation of Redmond, Washington, financial reportingprograms, customer registration programs, programs providing technicalsupport information, customer database applications, or application setmanagers.

[0039] During execution of the application program 158, a server 150communicates with the client node 110 over a transport mechanism (partof the server agent 160). In one embodiment, the transport mechanismprovides multiple virtual channels and one of the virtual channelsprovides a protocol for transmission of graphical screen data from theserver node 150 to the client node 110. The server 150 executes aprotocol driver (part of the server agent 160) that intercepts graphicaldisplay interface commands (generated by the application program 158 andtargeted at the server's operating system 156) and translates them intoa protocol packet suitable for transmission over the transportmechanism.

[0040] In one embodiment and with reference to FIG. 2, the server agent160 intercepts a graphical display interface command (step 202) issuedby the application program 158 and directed to the operating system 156,encodes graphical data associated with the command and the applicationprogram 158 (step 204), compresses the encoded graphical data (step206), creates a transport protocol packet incorporating the interceptedcommand and compressed, encoded data (step 208), and transmits thetransport protocol packet to the target client node 110 (step 210).Graphical data refers to any type of data that can be displayed on thedisplay 128, such as bitmaps, glyphs, coordinates, list of lines,strips, etc. Hereinafter, descriptions of techniques which are appliedto particular types of graphical data or graphical objects can beconstrued to apply to all types of graphical data.

[0041] More particularly, a protocol driver of the server agent 160intercepts the graphical display interface commands (step 202). In oneembodiment the protocol driver intercepts high-level applicationprogramming interface calls made by the application program 158. In oneembodiment, the protocol driver intercepts GDI commands provided byWINDOWS-based operating systems. In another embodiment, the protocoldriver intercepts QUICKDRAW commands, provided by MAC/OS-based operatingsystems. In still other embodiments, the protocol driver interceptslower-level commands, such as interface calls to the device driverassociated with the graphical display of the server node 150. In stillother embodiments, the graphical display interface command may beduplicated rather than intercepted. In these embodiments, the display ofthe application program output data on the server 150 and the client 110will be substantially similar, if not identical. The protocol drivercreates a transport protocol packet based on the intercepted graphicaldisplay command (step 208). The protocol driver may use a protocolcommand set that contains an analogous command for each possibleintercepted display interface command. Alternatively, the protocoldriver may use a command set that is a subset of the intercepted displayinterface commands. In other embodiments, the protocol command set maybe a superset of the possible intercepted display interface commands.

[0042] For each intercepted GDI command and affected graphical data thatmust be transmitted to the client agent 118 in order to display theoutput of the application program 158 on the client's display 128, theserver agent 160 queues the related protocol command together with aunique key 412 (FIG. 4) associated with the affected graphical data. Theunique key 412 describes and uniquely identifies the graphical data. Inone embodiment the unique key is generated by performing a 64 bit cyclicredundancy check (CRC) on the graphical data. In another embodiment, the64 bit CRC is generated as two 32 bit CRCs, performed on the graphicaldata sequentially and in opposite directions. Other methods ofgenerating a unique key associated with graphical data, as known to oneskilled in the art, may also be used. The protocol command and theunique key 412 are queued at the server 150, together with othercommand/unique key sets. The queued commands are then scanned toascertain whether later-issued commands (i.e., those near the back ofthe queue) supercede earlier-issued commands (i.e., those closer to thefront of the queue). If the earlier-issued commands are superceded, thenthe server agent 160 removes them from the queue and thus avoids havingto transmit unnecessary data to the client agent 118

[0043] For example, a series of commands to display the next page ofdata issued in quick succession need not be displayed individually atthe client 110 All that need be displayed is the last display pageissued, since this last page of data will overwrite all previous pagesof data. By removing the unnecessary commands and unique keyscorresponding to the earlier pages from the server's queue, the serveragent 160 substantially reduces the amount of data that must beprocessed and transmitted.

[0044] In one illustrative embodiment, only unnecessary commands andrelated data pertaining to opaque operations are removed from theserver's queue. For example, if a previously-queued command involvingthe creation of a rectangular region having coordinates 10,10,50,50 isfollowed by a more recently-queued command involving the creation of alarger/covering rectangular region having coordinates 0,0,100,100, thenthe previously-queued command and related data is deemed unnecessary andis removed from the queue.

[0045] Table 1 below lists an exemplary set of protocol commandsprovided in one embodiment of the invention. TABLE 1 PROTOCOL COMMANDSProtocol Command Purpose INIT Initializes the protocol connectionSET_MOUSE_POINTER Instructs the client agent 118 to form a mouse pointerHIDE_MOUSE_POINTER Instructs the client agent 118 to hide a mousepointer CHANGE_TEXT_MODE Instructs the client agent 118 to change themode of output text CHANGE_TEXT_COLOR Instructs the client agent 118 tochange the foreground color of output text CHANGE_TEXT_BACKGRND_COLORInstructs the client agent 118 to change the background color of theoutput text NEW_SOLID_BRUSH Instructs the client agent 118 to update thecurrent brush setting using a new solid color NEW_PATTERN_BRUSHInstructs the client agent 118 to change the brush patternNULL_CLIP_REGION Informs the client agent 118 that the current region isnot clipped SIMPLE_CLIP_REGION Informs the client agent 118 that thecurrent region is clipped along one side COMPLEX_CLIP_REGION Informs theclient agent 118 that the current region is clipped along more than oneside CHANGE_PALETTE Instructs the client agent 118 to change its colorpalette TEXTOUT Instructs the client agent 118 to output text,represented as one or more screen glyphs DRAW_PATH Instructs the clientagent 118 to draw a simple or complex path, such as a line or beziercurve BITBLT Instructs the client agent 118 to draw a particularbitmapped object. BITBLT_TRICK Instructs the client agent 118 to drawthe entire bitmap PARTIAL_BITBLT_TRICK Instructs the client agent 118 todraw a portion of the bitmap SRC_TO_SRC_BLT Instructs the client agent118 to copy a particular portion of the drawing surface to another ofthe same surface SOLID_FILL Instructs the client agent 118 to fill atarget region using an identified color SOLID_FILL_NO_CLIPPING Instructsthe client agent 118 to fill a target region without clippingSOLID_FILL_NEW_COLOR Instructs the client agent 118 to fill a targetregion using a color that is different from the color last usedSOLID_FILL_NEW_COLOR_NO_CLIP Instructs the client agent 118 to fill atarget region using a color that is different from the color last used,without clipping CACHE_NEW_OBJ Sends data representing a new object tothe client agent 118 for storage in the client's cacheCACHE_NEW_OBJ_INCOMPLETE Sends partial data of a new object to theclient agent 118 for storage in the client's cache CACHE_EXTEND_OBJSends additional data relating to a previously transmitted incompleteobject and indicates that the object is completeCACHE_EXTEND_OBJ_INCMPLT Sends additional data relating to a previouslytransmitted incomplete object and indicates that more data will followCACHE_WRITE_DISK_OBJECTS Instructs the client agent 118 to write itsobject cache, or a portion of its object cache to persistent storageCACHE_PURGE_MEMORY_CACHE Instructs the client agent 118 to purge itsobject cache, or a portion of its object cache CACHE_READ_DISK_OBJECTInstructs the client agent 118 to read data into its cache frompersistent storage START_STOPWATCH Instructs the client agent 118 tostart a timer and write the current time tick into an array elementSTOP_STOPWATCH Instructs the client agent 118 to determine thedifference between the time tick recorded by the START_STOPWATCH commandand the current time tick SAVE_SCREEN_BITS Instructs the client agent118 to save a rectangular screen area and to associate it with aparticular identifier RESTORE_AND_FREE_SCREEN_BITS Instructs the clientagent 118 to restore a particular rectangular screen area and to thendiscard the screen data FREE_SCREEN_BITS Instructs the client agent 118to discard specified data CREATE_SURFACE Instructs the client agent 118to form an off-screen surface compatible with the current session colordepth, associate the surface with an identifier, and associateattributes to the surface DELETE_SURFACE Instructs the client agent 118to delete a particular off-screen surface CHANGE_SURFACE Instructs theclient agent 118 to switch from a current destination surface to anotherspecified surface C2S_OSS_ERROR Informs the server agent 160 of an errorin a particular off-screen surface that makes it unavailableC2S_INVALIDATE_OSS Informs the server agent 160 that a particularoff-screen surface has become dirty, i.e., that it is not the same as aduplicate surface accessible to the server agent 160C2S_UPDATE_DISK_KEYS Instructs the server agent 160 to update its listof fuzzy keys C2S_ACK_DISK_READS Informs the server agent 160 thatobjects have been read into the client's cache C2S_NACK_DISK_READInforms the server agent 160 that objects have not been read into theclient's cache C2S_STOPWATCH_RESULT Sends the time differential betweenthe start/stop stopwatch commands to the server agent 160 C2S_ACK_lINTITInforms the server agent 160 that the protocol connection has beenestablished

[0046] In one embodiment, the protocol commands and associated graphicaldata are configured to maximize the repetition of information in theprotocol packet stream created by the protocol driver. In one aspect ofthe invention, the graphical data is first encoded (step 204) prior tosubjecting the encoded graphical data and/or protocol commands to acompression algorithm (step 206) that takes advantage of the repetitivenature of the protocol stream (e.g., a Lempel-Ziv compression protocolmay be used). In one embodiment and upon the first instance of thegraphical data in the client-server session, an indicia of the graphicaldata is transmitted together with the corresponding encoded data in theprotocol stream so as to reduce the amount of data transferred via thenetwork 140 for future instances of the graphical data occurring duringthe same client-server session. The compressed data is subsequentlybundled into the transport protocol packet and transmitted to the clientnode 110 (step 208) for decompression and interpretation.

[0047] In one illustrative embodiment, the application program 158executes a GDI command that instructs the server's operating system 156to draw a bitmap. The server agent 160 intercepts this GDI command (step202) and issues a BITBLT command to the client agent 118, whichinstructs the client agent 118 to display the application program'soutput data on the client's display 128 Prior to issuing the BITBLTcommand, the server agent 160 encodes the output data (step 204) andincludes the

[0048] encoded data, together with the BITBLT command in the compressedprotocol packet stream transmitted to the client agent 118 In oneembodiment, the data describing the bitmap 410 (FIG. 4) is encoded andcompressed (steps 204 and 206) to take advantage of some of theredundancies present in the bitmap 410. For example, the bitmap 410 maybe such that successive raster lines are redundant or adjacent pixelsare redundant.

[0049] In addition to encoding bitmaps or other discrete graphical dataelements based on their internal data redundancies, the invention alsoincorporates caching techniques that further improve the encoding andcompression of graphical data that has been previously encounteredduring a particular client-server session. For example, when aparticular bitmap is first encountered during a client-server session itis encoded as described above. For subsequent transmissions of thebitmap during the same client-server session, indicia of the bitmap canbe transmitted rather than the relatively larger, encoded bitmap. In oneembodiment, the indicia correspond to a location within the client'svolatile memory that contains the previously transmitted bitmap.

[0050] More particularly and with reference to FIGS. 3 and 4, the serveragent 160 generates the unique key 412 (step 330), as described above,and associates the unique key 412 with the graphical data (e.g., bitmap410). This association essentially defines the bitmap 410 and itsrelated data as a graphical object, which is subsequently registered,together with its unique key 412 and encoded data 414 (step 340), in adata structure 411 (e.g., a table) stored in the server's volatilememory 154 (e.g. the server's cache subsystem). In other embodiments,the bitmap-related data may be written into the server's persistentstorage 162.

[0051] Assuming that the protocol command (e.g., BITBLT) and associatedgraphical data are not superceded by later-issued commands (as describedpreviously), the server agent 160 dequeues (step 360) the BITBLT commandand the unique key 412 from the queue and searches the table 411 storedin the server's volatile memory 154 to determine if the unique key 412had been previously stored therein. If the unique key 412 is locatedwithin the table 411, then the server agent 160 determines that theencoded data 414 has been previously processed by the server 150. Theserver agent 160 then searches the table 411 for an index or handle 416associated with the encoded bitmap 414. If the index 416 is located,then the server agent 160 determines that the encoded bitmap 414 has notonly been processed by the server 150, but that it has also beenpreviously transmitted to the client agent 118. If the index 416 is notlocated within the table 411, then the server agent 160 determines thatthe encoded bitmap 414 has not been previously transmitted.

[0052] In one embodiment, the index 416 identifies a particular locationwithin the client's volatile memory 114, which stores thepreviously-transmitted encoded bitmap 414. In one embodiment, the clientagent 118 provides a snapshot of its available volatile memory 114 tothe server agent 160 upon initiating a client-server session with theserver agent 160. Once the session is established, the server agent 160thereafter controls the allocation of the client's available volatilememory 114. In this manner, the server agent 160 is able to allocate theclient's volatile memory 114 to the encoded bitmap 414 and to maintainthe index 416 as a pointer or reference to the allocated memory and asan identifier of the previously-transmitted encoded bitmap 414.

[0053] If the index 416 is located within the server's table 411, theserver agent 160 obtains the index 416 (step 370), compresses the index416 together with the related protocol command (step 380), and transmitsthe compressed data in a packet directed to the client agent 118 via thenetwork 140 In one embodiment, the index 416 is shorter in length (e.g.,16 bits) than the unique key 412 (e.g., 64 bits), and consequentlyconsumes less bandwidth when transmitted. Upon receipt and decompressionof the compressed packet, the client agent 118 accesses the particularmemory location in the client's volatile memory 114 that is specified bythe index 416 in order to obtain the appropriate graphical data 410. Theclient agent 118 subsequently manipulates the obtained graphical data410 in accordance with the command which accompanied the index 416 inthe compressed packet. In this manner, the invention avoids having toretransmit relatively large bitmap or other graphical data that havebeen previously encountered during a particular client-server session.

[0054] If the index 416 is not located within the server's table 411(and thus has not been previously transmitted to the client agent 118),the server agent 160 allocates a portion of the client's volatile memory114 for the bitmap 410 and forms the index 416 that identifies thememory location. The server agent 160 then stores the encoded bitmap 414(corresponding to the bitmap 410), the newly-formed index 416, and theunique key 412 associated with the encoded bitmap 414 in the server'stable 411 The server agent 160 subsequently compresses the encodedbitmap 414, the index 416, and related command and transmits thecompressed data in a packet directed to the client agent 118 Uponreceipt and decompression, the client agent 118 decodes the encoded data414 and stores the resulting bitmap 410 in the memory locationidentified by the index 416 and manipulates the resulting bitmap 410 inaccordance with the accompanying command. Alternatively, the clientagent 118 stores the encoded data 414 in the memory location identifiedby the index 416 and only decodes the encoded data 414 when necessary tofurther manipulate the resulting bitmap 410. In this manner, clientmemory is more efficiently used.

[0055] Although the invention has heretofore been described with theserver agent 160 controlling the client's memory allocation and indexformation, it is also possible that the client agent 118 perform thesefunctions. For example, when a bitmap is first encountered during aclient-server session and is encoded, compressed, and transmitted to theclient agent 118 (with the accompanying command, but without the index416), the client agent 118 can determine the particular location withinits volatile memory 114 to store the bitmap 410. In this embodiment, theclient agent 118 forms an index (not shown) that uniquely identifies theencoded bitmap and its corresponding memory location and transmits theindex to the server agent 160, which stores the index in the appropriatelocation in the server's table 411.

[0056] Continuing with the embodiment in which the server agent 160controls the allocation of the client's volatile memory 114, the clientagent 118 maintains a circular queue 421 of all indexes specified by theserver agent 160 in accordance with queue maintenance instructions alsoreceived therefrom. For example, the server agent 160 can direct theclient agent 118 to disassociate a predetermined block of indexes 417(e.g., corresponding to 128K of memory) from their respective graphicaldata in order to accommodate newly-encountered graphical objects whenthe client's volatile memory 114 is otherwise fully subscribed.Consequently, a subsequent new graphical object acquires apreviously-used index.

[0057] The server agent 160 also determines, according to predeterminedcriteria, whether to purge or save each graphical object affected by thereassignment of the indexes. The server agent 160 communicates thisdecision to the client agent 118 For example, if the server agent 160 isreassigning indexes 0, 1, 2, and 3, and determines that the graphicalobjects associated with handles 0 and 2 are to be deleted or moved intothe client's persistent storage 120 and the objects associated withhandles 1 and 3 are to be saved, the server agent 160 thus instructs theclient agent 118 to delete (or move) 0 and 2, and save 1 and 3 Theclient agent 118 then moves the index of each saved object to the tailof the circular queue. At least one index remains available at the queuetail for assignment to the moved object; that is, in one embodiment thehead and tail of the queue do not point to the same index in thecircular queue. Consequently, a moved object is both stored in adifferent physical memory location and associated with a differentindex.

[0058] The maintenance actions performed on the client's circular queue421 will typically become more frequent as the client-server sessioncontinues and the limited volatile memory 114 of the client 110 becomesfull of previously-transmitted bitmaps 410 or other graphical objects.Accordingly, the number of previously-transmitted graphical objects 422that are moved into the client's persistent storage 120 from volatilememory 114 increases. The present invention further enhances the cachingtechnique described above to search for indicia (see “fuzy key” below)of previously-transmitted graphical objects 422 that may be stored inthe client's persistent storage 120 when an index to that graphicalobject 422 is not found in the server's table 411. If such indicia isfound, then there is a strong likelihood that the previously-transmittedgraphical object 422 is still locally resident and accessible to theclient agent 118 and that the graphical object 422 may not thereforeneed to be retransmitted from the server agent 150 to the client agent110. It is important to note that locating such indicia provides astrong likelihood, but not a certainty, that the graphical object 422 islocally resident, because other programs or client agents which mayshare the same persistent storage 120 may have deleted the graphicalobject 422 from persistent storage 120 (e.g., to make room in thepersistent storage for a new object).

[0059] More particularly and with reference to FIGS. 4 and 5, if theserver agent 160 is unable to locate an index for a particular bitmap422 in the server's index table 411, the server agent 118 searches asecond table 418 (hereinafter “fuzzy database”) for an entry 419(hereinafter “fuzzy key”) associated with the bitmap 422 (step 520). Inone embodiment, each fuzzy key 419 is a combination of the unique key412 in the server's table 411 and the lengthsize of the associatedbitmap 422. The fuzzy key can be formed, for example, via an XOR(exclusive OR) of both 32-bit CRCs used to compute the 64-bit unique key412 in the server's table 411 and the length of the bitmap 422associated with the unique key 412. The fuzzy database 418 essentiallyprovides an initial prediction as to whether the persistent storage 120of the client 110 has a copy of the bitmap 422 stored therein. If thefuzzy database 418 does not contain the fuzzy key 419 associated withthe bitmap 422, then the client's persistent storage 120 probably doesnot have a stored copy of the bitmap 422 and the server agent 160 will,therefore, need to transmit an encoded version of the bitmap 422 to theclient agent 118, as previously described. On the other hand, if thefuzzy database 418 contains the fuzzy key 419 for the bitmap 422, thenthere is a strong possibility that the bitmap 422 is stored in theclient's persistent storage 120.

[0060] In more detail, when the server agent 160 does not find a fuzzykey 419 in the fuzzy database 418 that is associated with the bitmap422, the server agent 160 sends an encoded bitmap (corresponding to thebitmap 422) to the client agent 118 with a command that requests theclient agent 118 to store the bitmap 422 (or the corresponding encodedbitmap) in the volatile memory 114 (step 530). Also and as describedabove, the server agent 160 adds the fuzzy key 419 associated with thebitmap 422 to the fuzzy database 418 (step 540). Note that the order ofsteps 530 and 540 can be reversed, such that the fuzzy database 418 isupdated prior to the transmission of the command and the encoded versionof the bitmap 422. The client agent 118 subsequently copies the bitmap422 to a particular location within the client's volatile memory 114 asspecified by the server agent 160 (step 560).

[0061] In one embodiment, the server agent 160 does not wait for anacknowledgement from the client agent 118 that the encoded bitmapassociated with the bitmap 422 has been successfully received andprocessed. Rather, the server agent 160 assumes that the encoded bitmaphas been properly processed and continues to stream protocol commandswithout awaiting the return of an acknowledgement message from theclient agent 118. The server agent 160 does, however, track the numberof protocol commands issued since the last acknowledgement read. Theserver agent 160 blocks further transmission of protocol commands if thenumber reaches a predefined threshold. The threshold enables the clientagent 118 to establish an upper bound on the amount of memory needed toqueue incoming protocol commands during recovery mode operation, asdescribed in more detail below.

[0062] When a fuzzy key 419 is found in the fuzzy database 420, theserver agent 160 sends a command to the client 110 directing the clientagent 118 to copy the bitmap 422 associated with the fuzzy key 419 fromthe client's persistent storage 120 into the client's volatile memory114 (step 550). This command is accompanied not only by the fuzzy key419 which uniquely identifies the bitmap 422 of interest, but also bythe associated index generated by the server agent 160 (as previouslydescribed) in order to identify the specific location within theclient's volatile memory 114 that will receive and store the copiedbitmap 422. By copying the bitmap 422 into volatile memory 114 ratherthan just maintaining its location in persistent storage 120, theinvention can rapidly respond to several requests to draw the bitmap 422that are received in quick succession or without much intervening cacheactivity by copying the bitmap 422 on the occurrence of the firstrequest.

[0063] If the bitmap 422 is not currently stored in the persistentstorage 120 despite the presence of the corresponding fuzzy key 419 inthe fuzzy database 418, then the client agent 118 returns a message tothe server agent 160 (in response to the “CACHE_READ_DISK_OBJECT”command previously issued by the server agent 160) indicating that thebitmap 422 is missing. Upon receiving this message, the server agent 160retransmits the encoded bitmap data associated with the bitmap 422 tothe client agent 118 (step 530). Note that the server agent 160maintains a sequential list of all such CACHE_READ_DISK_OBJECT commandssent to the client agent 118 for which the server agent 160 has not yetreceived an acknowledgement so that the server agent 160 can properlyassociate a received acknowledgement with a particular bitmap 422 (so asto properly identify the encoded bitmap that needs to be transmitted).

[0064] Meanwhile, the client agent 118 enters a recovery mode in whichthe client agent 118 continues to read the incoming protocol stream, butdoes not process it. Instead, the client agent 118 builds a queue forreceiving the stream of commands that continue to flow from the server150 subsequent to the command that failed to find the bitmap 422 inpersistent storage 120. The client agent 118 continues to store thesecommands in this queue, in FIFO (first in, first out) fashion, until theencoded bitmap is received and successfully decoded to produce thebitmap 422. To recognize the missing bitmap 422 in the incoming streamof commands, the client agent 118 looks for graphical data accompanyinga recovery marker in the protocol stream. In one embodiment, therecovery marker is a particular pseudo-random number that is XORed withthe unique key of the bitmap 422 (i.e., the 64-bit CRC and 32-bitlength). The server agent 160 creates the recovery marker upon receivinga no-acknowledgement message from the client agent 118 When the bitmap422 arrives, the client agent 118 stores it in its volatile memory 114(step 560) (i.e., where the bitmap 422 would have been stored had theobject been initially present in the persistent storage 120) and beginsto process the commands in the queue. After processing all of thecommands in the queue, the client agent 118 resumes processing theincoming stream of protocol commands coming from the server 150 over thenetwork 140.

[0065] An advantage to the above-described recovery process is that itavoids the time delay penalty incurred by those cache systems that flushthe commands in the pipeline upon the occurrence of a cache miss. Also,the recovery process causes no disruption to the stream of protocolcommands nor any increase in the traffic on the channel by more than afew bytes beyond what the server 150 would have sent to the client 110had the server agent 160 initially known that the bitmap 422 was not inthe persistent storage 120.

[0066] In one embodiment, the present invention facilitates the locationof the bitmap 422 in the client's persistent storage 120 by storing thebitmap 422 in a file whose file name is formed by encoding the fuzzy key419 along with least-recently-used and/or least-frequently-usedinformation associated with the bitmap 422. In this manner, the filename itself provides the information desired and thus avoids having toopen and read the file. The encoding of the fuzzy key 419 as part of thebitmap's file name also enables the client agent 118 to rapidly extractthe fuzzy key information from the file name and send it to the serveragent 160 during an initial boot of the client 110 or at the start ofthe client/server session. The extracted fuzzy keys can be transmittedto the server 150 in an order that, for example, is based on theleast-recently-used and/or least-frequently-used information that alsoforms a part of the file name.

[0067] In one embodiment, fuzzy keys 419 are added to and deleted fromthe fuzzy database 418 in one of three ways. First, at the start-up ofthe client agent 118 or upon establishing a client-server session withthe server agent 160, the client agent 118 sends commands to the serveragent 160 to add a fuzzy key 419 to the fuzzy database 418 for eachappropriate bitmap 422 in the client's persistent storage 120 (the fuzzykeys accompany the client agent's commands). The client agent 118 canmaintain a send list that specifies an order (e.g., based onmost-recently-used or most-frequently-used information determined fromprior client-server sessions) for sending fuzzy keys at startup to theserver 150. The send list can be formed by the client agent 118 uponreading the directory of its persistent storage 120 and keeping arepresentation of the directory in volatile memory 114. In one aspect ofthe invention, the client agent 118 extracts fuzzy key information fromthe file name of the file, which contains the bitmap 422, when preparingthe representation of the directory. Second, at predetermined pollingintervals, the client agent 118 polls its persistent storage 120 todetermine those bitmaps that are currently in its persistent storage 120(e.g., by searching for file names containing a fuzzy key-likestructure) and then sends add and/or delete commands to the server agent160, as needed to update the server's fuzzy database 418 Third, when theserver agent 160 sends a command to the client agent 118 to write abitmap into the client's persistent storage 120, the server agent 160generates and stores a corresponding fuzzy key 419 in the fuzzy database418.

[0068] Transmission of bitmap data from the server 150 to the client 110can take a relatively lengthy period of time, particularly when thenetwork connection between the client and server is relatively slow. Inone embodiment, the invention adapts the server's operation to changingnetwork conditions by determining the relative speed of the network 140.For example, by dynamically assessing the bandwidth of the network 140,the server agent 160 can modify the encoding and compression techniquesused to process bitmap data in order to reduce bandwidth requirementswhen transmitting over a slow network. The invention processes columnsof bitmap data from left to right and primes the client cache 114accordingly. As the encoded bitmap 414 is being transmitted, thetransmission time can be monitored to determine the performance of thenetwork 140. If the transmission time exceeds a threshold value, theclient agent 118 can draw whatever bitmap data has already been storedin its cache 114 and display the remaining portions of the bitmap dataeither in real-time as it is received in the cache 114 or atpredetermined intervals. In this manner, a user of the client willrecognize that the client 110 is still operating on the bitmap data andbe able to discern that the client 110 is not in a hung or failed/errorcondition. By approximating the effective network bandwidth to theclient 110 and adapting the behavior of the server 150 accordingly, theserver agent 160 can, for example, instruct the client agent 118 to usea whole bitmap versus using a whole column mode of display for aparticular bitmap.

[0069] More particularly, the server agent 160 sends a Start_StopWatchcommand followed by encoded bitmap data 414 and a Stop_StopWatch commandto the client agent 118. The client agent 118 responds to theStart_StopWatch command by reading the current time tick and writing itinto a first array element. After the encoded bitmap 414 is received,the client agent 118 responds to the Stop_StopWatch command by readingthe current time tick and writing it into a second array element.Comparing the difference in time ticks between the Start_StopWatch andStop_StopWatch commands provides an estimate of the time that wasexpended to receive the encoded bitmap data. The resulting timedifference is then transmitted back to the server agent 160. Thisprocedure can be repeated to compute a smoothed, moving average fromwhich a reasonable estimate of the effective throughput of encodedbitmap data can be determined. The invention uses the throughputestimate in view of the size of the encoded data for a particular bitmapto decide whether to use whole bitmap mode (e.g., if it is likely totake less than 5 seconds) or use progressive whole columns (e.g., if itis likely to take more than 5 seconds). Note that although thisprocedure does not result in a 100% accurate throughput estimate, itdoes provide sufficient granularity to detect fast or slow networkconnections from which different display modes can be determined. Asimilar technique can be used to time large screen-to-screen copies atthe client end and to build up a similar moving average estimate of theclient's performance during these operations, which can then be used toslow a producer thread on the server 150 and prevent an overrun at thesource when the server 150 generates data faster than the client 110 canhandle it.

[0070] In one embodiment, the invention forms objects that aresufficient in size to contain the data of a whole cache column. Forexample, the maximum object size and the size of the cache can beselected by the server agent 160 upon startup of the client/serversession to be a function of the size of the connection window and itscolor depth, with larger sizes being allocated for the larger screensand depths. Further, current techniques for breaking up a bitmap intocolumns can result in columns of less than standard width at one or bothends of the bitmap. If these nonstandard columns are too narrow, thenthey may not be large enough to be retained in the client's persistentstorage 120 and thus will not be present at the client 110 duringsubsequent client/server sessions. The invention avoids this problem bymerging the narrow columns with neighboring columns so that only widecolumns are used and thus all parts of a splash screen, for example,will be contained in the persistent storage 120. In one embodiment,whole columns are then displayed in two phases. The first phase primesthe cache 114 with any missing data and the second phase draws thecolumn from the cache 114, which may involve using more than one cachedobject. The column is therefore displayed on the display screen 128 inan atomic fashion. As previously discussed, when the network connectionis slow, the columns can be displayed atomically, one-by-one, toreassure the user at the client 110 that the system is still working.Alternatively, the whole bitmap can be displayed atomically using asimilar technique when a fast network connection is detected.

[0071] The moving average estimate technique discussed above can also beused to mitigate longstanding overscroll problems in an application thatis being executed on a fast server and that is being displayed on arelatively slow client. The overscroll problem occurs, for example, whena user of the client 110 initiates a plurality of scroll commands suchas by clicking on the scroll bar (or dragging the scrollbar) with amouse 126 a number of times. As the scroll commands are sent to the fastserver, the server 150 performs the scrolling operations and returns theassociated data faster than the slow client 110 (or slow network 140)displays the results on its display screen 128. Consequently when theuser at the client 110 eventually views the desired screen position andwants to stop the scrolling (by stopping the mouse clicks and/orscrollbar dragging), the display will continue to scroll beyond thedesired screen position. This overscroll condition occurs because thefast server 150 has already processed the scroll commands and hastransmitted the appropriate data back to the client 110, but the datahas been queued at the slow client and has not yet been entirelyprocessed.

[0072] The invention mitigates this overscroll problem by reducing itsfrequency of occurrence. More particularly, the invention periodicallytimes selected scroll events at the server 150 (by scrolling the framebuffer) and at the client 110 (via the StopWatch commands discussedabove) to compute a moving average estimate of their respective speeds.In this manner, the server agent 160 estimates how long a particularscroll event will take (speed times the number of pixels involved) toprocess at the server 150 and how long the client 110 is expected totake and if the expected processing time at the client 110 is largerthan that of the server 150, the server processing is suspended by theappropriate time differential so as to keep the client 110 and server150 substantially in step. This approach results in many feweroverscrolls due to the time lag between the client 110 and server 150 ascompared to the number of overscrolls occurring when this approach isnot implemented.

[0073] In addition to manipulating bitmap data, the present inventioncan also remote a text string that is generated on the server 150 to thedisplay 128 of the distant client 110 With reference to FIG. 6, theserver 150 executes the application program 158, which generates a textstring that needs to be remoted to the client 110 The server agent 160intercepts the commands sent from the application program 158 to theserver operating system 156, which relate to the text string, and causesa display driver 612 of the server agent 160 to process the text-relatedinformation and to transmit a series of protocol commands 614 to theclient agent 118 for displaying a text string 416 on the client'sdisplay 128. For example, the application 158 performs procedure callsthat set the text color and the background color (here, “Hello!”).Execution of the TextOut procedure 618 by the application 158 triggersexecution of the DrvTextOut procedure 620 of the display driver 612 Asshown, the DrvTextOut procedure 620 defines the clip region, thebounding rectangle, and the text mode (including background and textcolor) by executing a sequence of corresponding procedures.

[0074] Execution of such procedures cause the display driver 612 to sendthe appropriate data and protocol commands (such as a subset of thoseprovided in Table 1) to the client agent 118 The protocol commands arereceived and processed by the client agent 118, which executesprocedures 614 that specify the clip region, the bounding rectangle, thetext mode, and the glyph associated with each text character to bedisplayed. Note that the protocol associated with a glyph object encodesnot only the glyph bitmap itself, but also its relative positioninginformation. The server 150 can also send other protocol commands to theclient agent 118 that direct the client agent 118 to store theassociated glyphs in volatile memory 114 and/or in the persistentstorage 120, as previously described. The client agent 118 creates abitmap corresponding to the size of the bounding rectangle thatsurrounds the text string and sets the relative positions of each glyphin its proper position within the bounding rectangle. Once the lastglyph is received and its position set, the client agent 118 instructsthe client operating system 116 to draw/render the bitmap on the displayscreen 128 of the client 110.

[0075] The present invention also supports several differentdescriptions of paths (a set of lines or curves used as part of thedescription of a line drawing operation or, in the case of a closedpath, to describe a region) including ellipses, beziers, segments,strips, and styles. Segments refer to line segments that, when combined,form a path. Similarly, segments can be further parsed into strips,which are portions of the line segment (corresponding to a series ofconsecutive pixels) that exhibit the same angle characteristics. Theencoding technique used to represent strips, for example, uses relativelocation information and quantized angles to characterize consecutivestrips in particular line segments, which not only minimizes the amountof graphical line data that needs to be encoded but also results in amore repeatable sequence of data that can be more efficientlycompressed. The increase in repeatability is particularly enhanced whenrepresenting strips that form a shape such as a rectangle or circle. Theprotocol used for strips also enables the client agent 118 to accuratelyreproduce a path independently and without knowledge of the algorithmused by the server agent 160 to convert the path into a sequence ofpixels.

[0076] With reference to the nonlimiting example in FIG. 7A, a path 700comprises two line segments 710, 720, which appear to be perfectlylinear. However, those skilled in the art recognize that line segmentsthat are not perfectly horizontal, vertical, or at a 45 degree diagonalmust be approximated by a corresponding series of strips when the path700 is displayed on a display screen 128 (FIG. 1). This approximation isnecessary, because the individual pixels on the display screen 128 aresurrounded by, at most, eight other pixels that are either perfectlyhorizontal, vertical, or at a 45 degree diagonal relative to each other,and thus any line segments 710, 720 that are at a different overallangle, must be approximated by a series of strips. Accordingly and withreference to the exemplary path 700 of FIG. 7B, segment 710 is comprisedof four strips 712, 714, 716, 718 where the individual pixels (depictedby square blocks) of each strip 712, 714, 716, 718 are organized at 45degree diagonals relative to each other, and segment 720 is comprised offive strips 722, 724, 726, 728, 730 where the individual pixels of eachstrip 722, 724, 726, 728, 730 are organized horizontally relative toeach other.

[0077] In one embodiment and with reference to FIGS. 7B and 8A, theserver agent 160 parses the path 700 into line segments 710,720, eachhaving a particular length and angle (step 810). An array of linesegments is then formed to store quantized segment information(hereinafter referred to as a quantized angle or “QA”) for each linesegment 710,720 (step 820). With reference also to FIG. 8B, rather thancomputing the absolute angle associated with a particular segment710,720, the server agent 160 instead determines in which of the sixteenquantized angles 880 the absolute angle lies. This quantizationtechnique determines in which sixteenth of the circumference the endpoint lies for a line segment starting at the center of a circle (aradius). This information can be expressed as the unsigned delta fromthe last segment specified in this context (mod 16, initialized to 0).This technique results in a more repeatable sequence (than if theabsolute angle was used), particularly when the segments form a shapesuch as a rectangle or a circle.

[0078] The server agent 160 then parses each line segment 710,720 intostrips (712, 714, 716, 718; 722, 724, 726, 728, 730 respectively) (step830). The length of each strip of a particular segment 710,720 issubsequently stored in a strip length array for further processing (step840). In one embodiment, the Windows NT algorithm, bLine, is used tobreak up the path 700 into an array of strips. The NT algorithmdetermines the quantized angle of each strip and segment so that thereis no ambiguity associated with segments or strips that lie on aboundary between quantized angle regions. The NT algorithm verticallyflips a line going up so that the line is always going down (the y valuekeeps increasing) and sets the flag, FL_FLIP_V. Similarly, a line goingto the left is flipped horizontally so that it always goes to the right(the x value keeps increasing) and sets the flag, FL_FLIP_H. As aresult, the array of strips output by this algorithm fall into a singlequadrant. In this particular embodiment, the server agent 160 undoes theflipping operation of the NT algorithm so that the array of strips areflipped back to their previous original quantized angles to ensure thatthe endpoint of a previous strip and the starting point of the nextconsecutive strip coincide so as to avoid having to send the startingcoordinate of the next consecutive strip.

[0079] The server agent 160 then encodes the beginning coordinate of thepath 700 together with the quantized angle data in the line segmentarray and pixel length data in the strip length array to form a protocolstream (step 850). The encoded data is then subjected to a compressionalgorithm to take advantage of the data redundancies in the protocolstream (step 860) and the compressed graphical line data is subsequentlytransmitted to the client agent 118 for decoding and further processing(step 870).

[0080] With reference to the exemplary illustration in FIG. 9 and thequantized angle information of FIG. 10, the line segment 710 comprisesfour strips 712, 714, 716, 718 positioned within quantized angle 1 Thebeginning coordinate of the path 700, which is transmitted from theserver agent 160 to the client agent 118 as part of the compressedgraphical line data, corresponds to the first pixel in strip 712(positioned at the origin of 880). The graphical line data furtherincludes the pixel length of each strip 712, 714, 716, 718 in thesegment 710 Since the segment 710 does not correspond to a perfectlyhorizontal, vertical, or 45 degree diagonal, each consecutive strip 714,716, and 718 in segment 710 is adjusted by a particular delta value asindicated in FIG. 10 in order to properly approximate the overall angleof the segment 710 In the present example, in order to get from the lastpoint in the previous strip 712 to the beginning point in the nextconsecutive strip 714, the client agent 118 displays strip 714 atsubstantially the same x value and at an increasing y value. Thisprocess is repeated for each strip in the segment 710 as well as for thestrips in segment 720, which are positioned within QA 15.

[0081] Accordingly and in one embodiment, the protocol stream generatedby the present invention when processing strips comprises the startingposition of the path, an index (if any) corresponding to a location inthe client's volatile memory 114 (FIG. 1) which may already store aparticular strip transmitted some time earlier in the client-serversession, the number of segments in the path, the quantized angle of eachsegment (from which the delta values reflected in FIG. 10 aredetermined), the length of each strip together with some controlinformation, and quantized angle delta information that is added to thequantized angle of a first or prior segment in order to inform theclient agent 118 of the quantized angle of subsequent segments. In thismanner, the invention avoids having to send coordinate and absoluteangle information for each strip in the path. Further, the stripprocessing technique discussed above facilitates the efficientcompression of graphical line data by leveraging the repetitiveness ofthe data in the uncompressed protocol stream.

[0082] The present invention can also be applied to bitmaps that aredisplayed in an on-screen surface, as well as to bitmaps in anoff-screen surface (e.g., that are stored in a video display adapter'smemory and/or in a pre-allocated section of the client's volatile memory114 that will be operated on by a graphics conversion library).Off-screen surfaces are frequently formed by applications, such asMicrosoft Word, that write bitmaps to the off-screen surface until thesurface is complete, at which time the off-screen surface is displayedon the display screen 128 of the client 110 in final form as anon-screen surface. Therefore, off-screen surfaces frequently provide thesource for on-screen surfaces.

[0083] In one embodiment and with respect to off-screen surfaces, theclient agent 118 informs the server agent 160 as to the amount of localvolatile memory 114 available for off-screen surfaces so that the serveragent 160 does not form an off-screen surface larger than the client'savailable memory. The server agent 160 subsequently forms an off-screensurface and transmits it to the client agent 118 with instructions tostore the surface in its volatile memory 114 The client agent 118selects and allocates a particular section of its volatile memory 114 tothe surface and stores the surface therein. Each off-screen surface hasa set of associated attributes, such as an identifier that uniquelyidentifies the surface, pixel format, dimensions, drawing attributes,protocol state, etc.

[0084] In one embodiment, in order to efficiently use the client'svolatile memory 114, the server agent 160 can issue commands to theclient agent 118, directing that certain operations be performed to theoff-screen surface only when the off-screen surface interacts with theon-screen surface (e.g., when updating the on-screen surface with bitmapdata from the off-screen surface). If the only interaction between theon-screen surface and the off-screen surface is a final copy to thescreen, then the operations can be targeted to the on-screen surface andthus reduce the amount of bitmap data that must be transmitted over thenetwork 140 Alternatively, the source of the on-screen surface can berestricted to either correspond to an off-screen surface stored remotelyat the server 150 or to an off-screen surface stored locally at theclient 110 Typical operations to the off-screen surface that arerequested by the server agent 160 include updating the bitmap in theoff-screen surface with updated text, line, or other data.

[0085] In one embodiment and with reference to FIG. 11, the server agent160 intercepts calls made to the server operating system 156, whichrelate to the creation of an off-screen surface, and the server agent160 forms a duplicate copy of the surface and stores it, along with itsattributes, in the server's local memory. The server agent 160 thenissues a command to the client agent 118 to form the off-screen surfacein the client's volatile memory 114 (step 1102). The command isaccompanied by an encoded representation of the off-screen surface ifthis is the first instance of the off-screen surface or by an index orfuzzy key if the off-screen surface has been previously transmittedduring the client-server session (step 1104). The server agent 160instructs the client agent 118 to incorporate the graphical dataidentified by the index/fuzzy key indicia or encoded surface into theclient's off-screen surface (step 1106). The client agent 118 respondsto the server agent's instructions by copying the identifiedsurfacebitmap to the client's off-screen surface (step 1108). If aBITBLT command issued by the server agent 160 fails, for example, toform a bitmap in the client's off-screen surface, such as when a memoryallocation error occurs, the client agent 118 sends a C2S_OSS_Errorcommand to the server agent 160 indicating the failed condition (step1110). In response to the error condition, the server agent 160 uses itslocal copy of the off-screen surface as the source for updating theclient's off-screen surface. The client agent 118 uses the informationreceived from the server's local copy to update the off-screen surfaceon the client 110 and to correct any corrupteddirty regions of theoff-screen surface. Alternatively, when an error condition isencountered, the server agent 160 transmits its local/duplicate copy ofthe off-screen surface (step 1112) to the client agent 118 and instructsthe client agent 118 to use the duplicate surface as the source forupdating the client's on-screen surface and to discard the corruptedoff-screen surface at the client (step 1114).

[0086] Although the present invention has been described with referenceto specific details, it is not intended that such details should beregarded as limitations upon the scope of the invention, except as andto the extent that they are included in the accompanying claims.

What is claimed is:
 1. A method of adapting a rate of processingactivity in response to changing network conditions, the methodcomprising the steps of: processing graphical data by a server agent ata first rate, the processed graphical data being addressed to a clientagent via a network coupled to the client and server agent; determiningthe network conditions of the network; and adjusting, by the serveragent, the rate of processing activity of the server agent from thefirst rate to a second rate in response to a change in the networkconditions.
 2. The method of claim 1 further comprising the steps of:transmitting the processed graphical data to the client agent;measuring, by the client agent, a time differential associated withtransmitting the processed graphical data, the time differentialcorresponding to the change in the network conditions; and selecting thesecond rate in accordance with the time differential.
 3. The method ofclaim 1 further comprising the steps of: processing the graphical dataat the first rate in accordance with a first encoding scheme; selectinga second encoding scheme in response to the change in the networkconditions; and processing subsequent graphical data in accordance withthe second encoding scheme.
 4. A method of adapting a processingactivity operating on graphical data in response to changing networkconditions, the method comprising the steps of: processing graphicaldata by a server agent using a first encoding technique, the processedgraphical data being addressed to a client agent via a network coupledto the client and server agent; determining the network conditions ofthe network; selecting, by the server agent, a second encoding techniquein response to a change in the network conditions; and processingsubsequent graphical data by the server agent using the second encodingtechnique.
 5. The method of claim 4 further comprising the steps of:transmitting the processed graphical data to the client agent;measuring, by the client agent, a time differential associated withtransmitting the processed graphical data, the time differentialcorresponding to the change in the network conditions; and selecting thesecond encoding technique in accordance with the time differential.
 6. Amethod of adapting a processing rate of a server in response to aperformance mismatch between the server and a client coupled to theserver via a network, the method comprising the steps of: processinggraphical data by a server agent; determining a first time periodassociated with processing the graphical data at the server agent;processing the graphical data by a client agent; determining a secondtime period associated with processing the graphical data at the clientagent; determining the time differential between the first and secondtime periods; and adjusting, by the server agent, the processing rate ofthe server in accordance with the time differential.
 7. The method ofclaim 6 wherein the first time period is determined by scrolling a framebuffer of the server.
 8. A system for adapting a rate of processingactivity in response to changing network conditions, the systemcomprising: a client agent; a network coupled to the client agent, thenetwork conditions being associated with the network; a server agentcoupled to the client agent via the network, the server agent processinga first graphical data at a first rate, the processed first graphicaldata being addressed to the client agent, wherein the server agentadjusts the processing rate from the first rate to a second rate inresponse to a change in the network conditions.
 9. The system of claim 8wherein a first encoding scheme is used to process the first graphicaldata at the first rate and a second encoding scheme is used to process asecond graphical data at the second rate.
 10. A system for adapting aprocessing activity operating on graphical data in response to changingnetwork conditions, the system comprising: a client agent; a networkcoupled to the client agent, the network conditions being associatedwith the network; a server agent coupled to the client agent via thenetwork, the server agent processing a first graphical data using afirst encoding technique and a second graphical data using a secondencoding technique in response to a change in the network conditions,wherein the server agent transmits the first and second graphical datato the client agent via the network.
 11. The system of claim 10 whereinthe change in the network conditions is detected from a timedifferential determined by the client agent.
 12. A system for adapting aprocessing rate of a server in response to a performance mismatchbetween the server and a client coupled to the server via a network, thesystem comprising: a client agent, the client agent processing graphicaldata at a client rate; a server agent coupled to the client agent via anetwork, the server agent processing the graphical data at a serverrate, wherein the server agent adjusts the server rate in response to adifference between the client and server rates.
 13. The system of claim12 wherein the server rate is determined by scrolling a frame buffer ofthe server.